A full two-way infrared telephone system exists as described in my copending U.S. patent application Ser. No. 619,803, filed June 12, 1984, now U.S. Pat. No. 4757553 entitled COMMUNICATION SYSTEM WITH PORTABLE UNIT. This patent application is incorporated herein by reference. The infrared telephone system uses a cordless handset that is portable as with rf cordless telephones. The infrared telephone system includes a base unit having an up-link infrared transmitter to send an audio FM modulated infrared carrier to the handset receiver and a base unit infrared receiver to detect an audio FM modulated infrared carrier back from a handset located infrared transmitter. Battery power of the portable handset is conserved by use of a power control circuit whereby the optical power from the handset, or down-link infrared transmitter, is kept at a minimum level needed to obtain good performance. This is achieved by sensing the level of the infrared signals at the infrared receiver in the base-unit and then varying the average frequency of the carrier applied in the base unit to the up-link infrared transmitter. This carrier frequency shift is sensed in the handset and used to vary the optical power from the handset completing a control loop. In the process the up-link carrier frequency is controlled at the center of the pass band of the handset infrared receiver.